Papers with transfer learning techniques
An Empirical Study on Cross-X Transfer for Legal Judgment Prediction (2022.aacl-main)
Copied to clipboard
| Challenge: | Cross-lingual transfer learning is understudied in legal NLP but not in legal Judgment Prediction (LJP). |
| Approach: | They explore cross-lingual transfer learning techniques on legal JP using a trilingual Swiss-Judgment-Prediction dataset and adapter-based fine-tuning. |
| Outcome: | The proposed methods improve the model’s performance by augmenting the training dataset with machine-translated versions of the original documents, using a 3 larger training corpus. |
DeepPavlov 1.0: Your Gateway to Advanced NLP Models Backed by Transformers and Transfer Learning (2024.emnlp-demo)
Copied to clipboard
Maksim Savkin, Anastasia Voznyuk, Fedor Ignatov, Anna Korzanova, Dmitry Karpov, Alexander Popov, Vasily Konovalov
| Challenge: | Open-source framework for using NLP models is released for non-experts . complexity of building, fine-tuning and deploying state-of-the-art models remains a barrier . |
| Approach: | They present DeepPavlov 1.0, an open-source framework for using NLP models . the framework is based on PyTorch and supports HuggingFace transformers . |
| Outcome: | The DeepPavlov 1.0 framework is designed for practitioners with limited knowledge of NLP/ML. |
Named Entity Recognition without Labelled Data: A Weak Supervision Approach (2020.acl-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) performance often degrades when applied to target domains that differ from the texts observed during training. |
| Approach: | They propose a method to learn NER models in the absence of labelled data through weak supervision by using a broad spectrum of labelling functions to automatically annotate texts from the target domain. |
| Outcome: | The proposed approach improves on two English datasets and shows that it improves by 7 percentage points on entity-level F1 scores compared to an out-of-domain neural NER model. |
Should I try multiple optimizers when fine-tuning a pre-trained Transformer for NLP tasks? Should I tune their hyperparameters? (2024.eacl-long)
Copied to clipboard
| Challenge: | NLP research has explored different neural model architectures and sizes, datasets, training objectives, and transfer-learning techniques. |
| Approach: | They propose to use a variant of Stochastic Gradient Descent (SGD) to select among numerous variants, often with minimal or no tuning of the optimizer’s hyperparameters. |
| Outcome: | Experiments with five GLUE datasets, two models and seven popular optimizers show that tuning just the learning rate is as good as tuning all the hyperparameters. |
XL-AMR: Enabling Cross-Lingual AMR Parsing with Transfer Learning Techniques (2020.emnlp-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a popular formalism of natural language. |
| Approach: | They develop a cross-lingual AMR parser that can be trained on the produced data . they use transfer learning techniques to produce automatic AMR annotations across languages . |
| Outcome: | The proposed parser significantly surpasses those reported in Chinese, German, Italian and Spanish. |
Low-Resource Comparative Opinion Quintuple Extraction by Data Augmentation with Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Comparative Opinion Quintuple Extraction (COQE) aims to predict comparative opinion quintuples from comparative sentences. |
| Approach: | They propose a low-resource approach to extract comparative opinion quintuples from comparative sentences . they propose augmentation using ChatGPT and a data-centric approach . |
| Outcome: | The proposed approach improves the existing pipeline-based method and achieves state-of-the-art results. |
Towards the First Machine Translation System for Sumerian Transliterations (2020.coling-main)
Copied to clipboard
| Challenge: | Sumerian cuneiform script was invented more than 5,000 years ago and is one of the oldest in history. |
| Approach: | They propose to translate Sumerian texts into English automatically using supervised, phrase-based, and transfer learning techniques. |
| Outcome: | The proposed method accelerates the costly and time-consuming manual translation process and helps researchers better explore the relationships between Sumerian and Mesopotamian culture. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese (P19-1)
Copied to clipboard
| Challenge: | Text-based transfer learning techniques can be used to perform downstream tasks. |
| Approach: | They propose to use text-based transfer learning techniques to pre-train a language model in an unsupervised manner and leverage them to perform effective on downstream tasks. |
| Outcome: | The proposed model performs better than task-specific models trained on 3 times as much data and is as effective for language modeling pre-trained on 1/30 of the data. |